Hi @Dr-Irv , thank you for drafting the plan.

Current plan

I think I prefer where if you are doing an operation like subtraction where it will sometimes work and sometimes not work, and the inferred type of one of the operands is Series[Any], then we detect that as a typing problem. But we need to be selective.

I think we have to be selective here, and probably disallow subtraction with untyped Series when the other argument is known to be time related (Timestamp, Timedelta and associated Series) or is a string or Series[str].

I would like to summarise this typing plan as following:

1. When the calculation can give a runtime error, typing shows an error or Never
2. Certain cases are exceptions

Timestamp and Timedelta: permissive or forbidding

With this typing plan, I have the following examples in my mind:

1. Series[Any] (int) - TimestampSeries -> error at type checking, error at runtime
2. Series[Any] (Timestamp) - TimestampSeries -> error at type checking, TimedeltaSeires at runtime

As a user I probably do not want the static type checker to aggressivly point out a potential problem. When the stub is less permissive and more forbidding, the static type checker becomes more aggresive. It seems better to me to allow both cases at the stage of static type checking, otherwise the user may need to manually ignore the type checker in many cases.

int: exceptions to the plan

"We need to be selective" is important in the plan, because we also have

1. Series[Any] (int) + Series[int] -> Series[Any] at type checking, Series[int] at runtime
2. Series[Any] (str) + Series[int] -> Series[Any] at type checking, error at runtime

Currently we are happy with the stub giving us Series[Any] for adding Series[Any] to Series[int]. This is an exception, which may potentially confuse the user.

Proposing a consistent plan

I would like to propose a new typing plan as following:

1. When the calculation gives several typing results or a runtime error, typing shows Series[Any]
2. When the calculation gives one typing result, say Series[R], or a runtime error, typing shows Series[R]
3. When the calculation always gives a runtime error, typing shows an error or Never

With this typing plan, the previous examples give different results:

1. Series[Any] (int) - TimestampSeries -> TimedeltaSeries at type checking, error at runtime (TimedeltaSeries is the only possible result that is valid, so unfortunately the type checker does not cache the potential problem here)
2. Series[Any] (Timestamp) - TimestampSeries -> TimedeltaSeries at type checking, TimedeltaSeires at runtime
3. Series[Any] (int) + Series[int] -> Series[Any] at type checking, Series[int] at runtime (no exceptional rule in the plan)
4. Series[Any] (str) + Series[int] -> Series[Any] at type checking, error at runtime (Series[float], Series[int] etc. are possible valid results, so unfortunately the type checker does not cache the potential problem here)

Further examples:

1. Series[Any] (int) + Series[str] -> Series[str] at type checking, error at runtime (Series[str] is the only possible result that is valid, so unfortunately the type checker does not cache the potential problem here)
2. Series[Any] (str) + Series[str] -> Series[str] at type checking, Series[str] at runtime
3. Series[Any] * TimestampSeries -> error / Never at type checking, error at runtime (Timestamp is consistently not multiplicative)

Thank you for reading the lengthy explanation. What do you think?

Thank you for reading the lengthy explanation. What do you think?

The challenge here is the issue of wide vs narrow types. See https://github.com/pandas-dev/pandas-stubs/blob/main/docs/philosophy.md#narrow-vs-wide-arguments for some writeup I did about that.

Let's consider this example from your list:

Series[Any] (int) - TimestampSeries -> TimedeltaSeries at type checking, error at runtime

In what is in main today, the following code works as you describe there, i.e., the type checker infers that result is TimedeltaSeries, but it fails at runtime.

si = pd.DataFrame({"a": pd.Series([1,2,3])})["a"]
st = pd.Series(pd.date_range("1/1/2005", "1/3/2005"))
result = si - st

I think we do a better service to users if we actually catch this via typing, i.e., for Timedelta, TimedeltaSeries, Timestamp, TimestampSeries, str and Series[str], if they are in a binary operation with a Series[Any] (either before the operator or after the operator), the type checker reports an error. That's telling the user "We don't know how to handle a generic series with another operand that has a specified type", but we are limiting the types we do that with to just the ones I mentioned.

This makes the user then cast the variable si above to Series[int] (in which case we catch the failure), and know it will possibly fail at runtime.

Let's also consider this example:

st = pd.Series(pd.date_range("1/1/2005", "1/3/2005"))
sd = pd.DataFrame({"a": [pd.Timedelta("1 day"), pd.Timedelta("2 days"), pd.Timedelta("3 days")]})["a"]
result = st - sd

In this case, if we adopt my proposal, the type checker would say that st - sd is invalid. But the type of sd is partially unknown, so we are then suggesting that the user do:

result = st - cast("pd.Series[Timedelta]", sd)

which is telling the type checker "I know this is a series of timedeltas"

I'm choosing what I consider to be a happy medium here between your proposal, and something that would be too narrow (e.g., disallowing Series[Any].__sub__(Series[Any])), by suggesting that if we know the types of ONE of the operands, but not the other, we try to catch the error via static typing.

I should say that the current behavior in the stubs is from 3 years ago when we first inherited the project from something MIcrosoft had started, and now that I have more experience with typing, as well as using the stubs in my own code, I've come around to trying to find more things with static type checking if we can find them, then not.

So the summary of my proposal is (with respect to Series) for binary operators a X b, where X is the operator:

1. If a and b are fully typed, we figure out the result, and if it is an invalid calculation, we catch it.
2. If a is Series[Any] and b is fully typed, we say that is an error.
3. If a is fully typed, and b is Series[Any], we say that is an error.
4. If a and b are not fully typed (i.e., one is Series[Any] and the other is Any or Series[Any], we accept the calculation in typing and don't report an error.

Let me know your thoughts on that.

Hi @Dr-Irv , thank you again for the detailed reply.

TimestampSeries

1. If a and b are fully typed, we figure out the result, and if it is an invalid calculation, we catch it.
2. If a is Series[Any] and b is fully typed, we say that is an error.
3. If a is fully typed, and b is Series[Any], we say that is an error.
4. If a and b are not fully typed (i.e., one is Series[Any] and the other is Any or Series[Any], we accept the calculation in typing and don't report an error.

I am concerned with the compatibility of 2+3 and 4.

• We can make Series[Any] - TimestampSeries not explicitly typed, so that the type checking shows an error.
• However we already have Series[S1] - Series -> Series[Any]
• When we remove TimestampSeries, Series[Any] - Series[Timestamp] will default to ``Series[S1] - Series, giving Series`.
• Along this practice, we would need to explicitly set Series[Never] - TimestampSeries -> Never for now, and after the removal of TimestampSeries, Series[Never] - Series[Timestamp] -> Never
• Alternatively, we will not be able to remove TimestampSeries.

In my proposal, I would rather allow Series[Any] - TimestampSeries as well as Series[Any] - TimedeltaSeries, leaving the user to check the potential error at runtime. We can either give Series[Any] in both cases, or, using my plan, give TimedeltaSeries in the first case.

We have already done the same to allow Series[Any] - Series[int], giving Series[Any], which could also fail at runtime if Series[Any] actually contains str.

In my proposal, I would rather allow Series[Any] - TimestampSeries as well as Series[Any] - TimedeltaSeries, leaving the user to check the potential error at runtime. We can either give Series[Any] in both cases, or, using my plan, give TimedeltaSeries in the first case.

Or we can catch it in type checking??

We have already done the same to allow Series[Any] - Series[int], giving Series[Any], which could also fail at runtime if Series[Any] actually contains str.

Yes, this latter point is quite valid. Here's a possible modification of my proposal: (with respect to Series) for binary operators a X b, where X is the operator:

1. If a and b are fully typed, we figure out the result, and if it is an invalid calculation, we catch it.
2. If a is Series[Any] and b is Timestamp, Series[Timestamp], Timedelta, Series[Timedelta], str, Series[str], we catch that as a typing error.
3. If a is Timestamp, Series[Timestamp], Timedelta, Series[Timedelta], str, Series[str], and b is Series[Any], we say that is an error.
4. If a and b are not fully typed (i.e., one is Series[Any] and the other is Any or Series[Any], we accept the calculation in typing and don't report an error.

Is this possible?

I tried to implement the plan in e0b5b59. The str part proves to be a bit tricky. I would like to leave it out for now.

I tried to implement the plan in e0b5b59. The str part proves to be a bit tricky. I would like to leave it out for now.

So should I review now?

So should I review now?

Yes please. I believe I have implemented the ideas in #1343 (comment). I would like to follow the proposal there.

There's been another issue with the overloads that has caused the dev version of mypy to fail on the current main. I discovered the problem, and if you look at this set of diffs, it fixes the problem, working with both mypy 1.17.1 and the dev version.

Can you incorporate these changes into your PR?

9c972d3...Dr-Irv:pandas-stubs:d3b8f164b2dc469146d43d49fd73eabc7ebe643a

Hi @Dr-Irv , thank you for asking me, I am honoured.

If I were in a the position to release a new version, I would do so now and save dropping TimestampSeries and TimedeltaSeries for the next release. The changes to arithmetic operations have already been significant, and it would be good to get some feedback.

Hi @Dr-Irv , thank you for asking me, I am honoured.

If I were in a the position to release a new version, I would do so now and save dropping TimestampSeries and TimedeltaSeries for the next release. The changes to arithmetic operations have already been significant, and it would be good to get some feedback.

It's done. Just released 2.3.2.250827